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ABSTRACT 

Several alternative indexes of item usefulness were 
empirically compared. They were; (1) item discrimination based on 
high and low groups on a postinstructional measure, (2) shift in item 
difficulty from a pre- to a postinstruction measure, and (3) item 
discrimination based on pre- and posttest performances. A typical 
classroom final examination was administered to 28 students on their 
first day in a multivariate statistics course and again at the end of 
the term. The results, using the various indices, indicated that 
shift in item difficulty from pre- to postinstruction yielded data 
significantly more similar to the pre-post, discrimination index than 
dlid the high-low group posttest discrimination index. It is 
recommended that the conceptually more ideal pjre- to postinstruction 
discrimination index be used even when instruction is not performance 
oriented. (Author/JS) 
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A Comparison of Traditional Item Analysis Selection 
Procedures with Those Recommended for Tests 
Dosignea to Measure Achievement Following 
Performance Oriented Instruction 

G,C, Helmstadtor, Arizona Stato University, Tempe, Arizona 

Several alternative indexes of item usefulness were empirically comparedi 
l) item discrimination based on high and low groups on a post instructional 
measure; 2) shift in item difficulty from a pre to a post instruction 
measure; and 3) iter, discrimination based on pre and post test perfor- 
mances T 
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Shift in item difficulty from pre to post instruction produced results 
significantly more similar to the pre-post discrimination index than did 
the high- low group post test discrimination index. It was recommended 
that the conceptually more ideal pre to post instruction discrimination 
index be used even v?hen instruction is not performance oriented. 
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A Comparison of Traditional Item Analysis Selection 
Procedures with Those Recommended for«Tests 
Designed to Measure Achievement Following 
Performance Oriented Instruction 

G.C, Helmstadter, Arizona State University, Tempe, Arizona 

Introduction 

Popham and Husek (1969) have stated that traditional item analysis pro- 
cedures may not be useful for constructing achievement tests in situations 
where the instruction has been performance oriented, Rahmlow, Matthews 
and Jung (1970) have further argued that in such criterion referenced 
situations the homogeneity of post instruction performance may be so 
great when the instruction has beon successful that discrimination indexes 
may not be useful in the traditional sense. Rather, these latter authors 
suggest that the function of a discrimination index in a criterion refer- 
enced situation is primarily that of indicating the homogeneity of the 
item with respect to the specific instructional objective measured. Fur- 
ther, as an alternative to the traditional interpretation of a discrimina- 
tion index, these latter authors focus attention on a shift in item diffi- 
culty from a non-instruction to a post-instruction measure. 

Unfortunately , the arguments offered against the use of traditional pro- 
cedures have assumed that discrimination indexes must be based on a com- 
parison of high and low performers on a post instructional measure and 
generally ignored the possibility that such an index might be based on a 
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difference in response to the item from a pre instruction to a post in- 
struction measurement. Thus it is entirely possible that the alternative 
procedure often proposed by advocates of criterion referenced measure- 
ment of using the change in item difficulty frora pre to post instruction 
measurement may be less efficacious in selecting items for achievement 
measures than the traditional item discrimination index applied in the 
more ideal way of contrasting responses before and after learning. 

The purpose of this study was, therefore, to empirically compare the fol- 
lowing three alternative indexes of item usefulness t 1) the item discri- 
mination based on high and low groups on a post instructional measure t 
2) the shift in proportion of subjects getting the item correct from a 
pre to a post instructional measure) and 3) the item discrimination based 
on pre and post test performances, 

I Method 

The procedure followed was to administer a typical classroom final exam- 
ination the first day of class to 28 students enrolled in a multivariate 
statistics course and again to the same subjects at the end of the term. 
The traditional item indexes of difficulty and of discrimination were com- 
puted in the usual uay&t both the pre instructional and the post instruc- 
tional responses to the item. An additional item discrimination index was 
then computed for each item comparing post instructional responses with 
pre instructional responses. Finally, the change in item difficulty from 
pre to post instruction administrations was determined for each item. 
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The extent to which the three different recommended item selection pro- 
cedures produced similar results was determined by computing the inter- 
correlations among the three indexes over the 59 items used in the exam- 
ination. 

Using a selection criterion of ,20 and up for the discrimination indexes 
and of ,10 or more increase in the proportion who got the item right on 
the post test as compared with the pre test, cross-tabulations were made 
to determine how many of the items selected and rejected by the more 
ideal procedure were similarly selected and rejected by the traditional 
high- low group post test discrimination index and by the sh if t in pro- 
portion correct index advocated by many persons using criterion referenced 
measures. 



Results 

A statistically significant (P<»001) increase in mean test scores for 
pre to post instruction of 19,2 and a change in average item difficulty 
from ,28 to ,59 indicated that changes in performance on the measure used 
did indeed occur. 

The correlation between the two discrimination indexes was found to be 
,47; that between the pre to post discrimination index and the shift 
in proportion correct was .78; and that between the post test high- low 
discrimination index and the shift in difficulty was .29. The difference 
between the correlations of .78 and .47 as determined by a formula appro- 
priate for situations where the correlations are both calculated on the 
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mbm subjects ms statistically significant (,001<P<,005). 

The cross tabulations indicated that of the 59 items included in the 
examinat ion, 67 percent would have been similarly selected and rejected 
by all three indexes! 71 percent would have been similarly selected or 
rejected by the pro to post discrimination index and the high-low dls- 
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crimination index) and 9® percent would have been simi larly selected or 



rejected by the pre to post discrimination index and the shift in pro- 
portion correct index. 



Discussion 

These data clearly confirm the contention of those who have urged cau- 
tion in using traditional item analysis procedures in criterion refer- 
enced situations. The results further suggest that even when the in-* 
struotion is not performance oriented the use of a high-low group discrim- 
ination index obtained cn a post instruction measure may not provide an 
appropriate index for the selection of items to be used in constructing 
an achievement test. 



While the use of the change in item difficulty from pre to post instruc- 
tion can lend to the selection of nearly the same set of item3 as the use 
of a pre to post instruction discrimination index, there seems to be no 
advantage to the use of shift in proportion who get the item correct over 
the more conceptually satisfying pre to post instruction discrimination 
index since both require two separate administrations of the pool of items 
from which the selection is to be made. 
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